PVLF 2025

Global Publishers Block AI Bots; Indian Media Joins Push

Top global publishers are blocking AI bots. Indian news media demands fair revenue sharing and legal safeguards to stop AI data theft and exploitation.
on Jul 03, 2025

Large international publishers are increasingly blocking AI web crawlers by default, backed by Cloudflare's move. Even in India, publishers are seeking equitable revenue sharing as well as legal safeguards against data theft by AI companies.

Following Cloudflare CEO Matthew Prince's dire warning to publishers, revealing how Google and artificial intelligence (AI) bots are redefining the internet in a manner that can "completely destroy the foundations of the open web", there is an emerging global momentum against unauthorised scraping of journalistic content by AI firms. And Indian digital news publishers have again called for equitable revenue sharing in India as well.

In a recent historic move in the US and UK, big publishers have started blocking AI web crawlers by default, supported by a new program from Cloudflare. Over a dozen international news organisations – such as Associated Press, The Atlantic, Sky News, Time, Buzzfeed, Conde Nast, and DMGT – are included in this initiative that aims to safeguard original journalism from unauthorised AI use.

Prince had indicated that Cloudflare noticed six months ago that 75% of searches on Google concluded without a single click. That percentage might be as large as 90% today. This is because Google more and more responds to questions on its own site at the top of the user's search, increasingly so since Google introduced its Overview AI feature, discouraging users from clicking through articles lower in the list.

Consumers hardly check original sources, and hence rely on AI-generated summaries. That implies creators lose reach as well as revenue, and reasons for publishing online disappear. And, AI gets to pick free content off publishers' sites.

What this means for India

Cloudflare's news that it is now blocking AI scrapers by default on all new sites marks a sea change in the industry. Website owners, whether publishers or not, now have full control over which AI spiders they permit – and on what conditions. This returns control to creators and enables them to negotiate with licensees or block unauthorized access entirely.

The global shift also has significant implications for India. Several Indian publishers have come forward to state that their news content, laboriously produced by human reporters and editors, is being used without permission or payment to train commercial AI models. Some of them have already begun to pool efforts and discuss legal recourse against such practices.

Indian publishers, led by industry associations like the Digital News Publishers Association (DNPA), have been protesting vehemently against what they call "unauthorised data theft" by AI makers.

"The India situation is becoming more and more untenable," opined a DNPA spokesperson. "While the world players are awakening to the need for permission and just compensation, Indian news content continues to be exploited freely without negotiation or protection. We urge the Government of India to take prompt action to take such measures against such unauthorised rampant data scrapping."

What has Cloudfare suggested?

Significantly, Cloudflare's system will also mark AI bots according to their intent – training, inference, or search – allowing content owners to be better informed. The firm is also testing a "pay-per-crawl" system, which would allow publishers to charge AI companies directly when thebots visit their sites.

This shift reflects growing concern over the impact of AI on media economics. According to Cloudflare data, OpenAI’s GPTBot alone accounted for 30% of all AI-related scraping in May 2025, up from 5% just a year earlier. Other aggressive scrapers include Meta’s External Agent and Anthropic’s ClaudeBot.

Top media executives have hailed this as a turning point. "This is a game-changer for publishers," Conde Nast CEO Roger Lynch said. "When AI firms can no longer appropriate anything they like for free, it opens the door to sustainable innovation founded on permission and partnership."

In India, though, the lack of an unambiguous regulatory framework or licensing model for AI utilization of copyrighted material places local publishers at a clear disadvantage. In contrast to the US and UK, where publishers enjoy legal recourse and increasing technological infrastructure, Indian creators of content are most frequently left vulnerable.

DNPA and other digital publishers have urged the Ministry of Electronics and IT (MeitY) and the Ministry of Information and Broadcasting (MIB) to:

Acknowledge unauthorised AI scraping as copyright violation
Make consent-based access mandatory for AI training
Facilitate the establishment of an Indian licensing system, perhaps patterned on those now being developed in the West
Offer technology tools, alongside companies such as Cloudflare, to enable smaller publishers

"India can be an AI world leader – but at the expense of crushing the rights of its own creators," added a senior editor with a national digital newspaper. "We need to innovate responsibly, with legislation that respects original content and upholds public trust."

As the global conversation shifts toward ethical AI development and fair compensation, India's digital content ecosystem is at a crossroads. The question now is whether India will follow the global trend or be left behind in safeguarding the rights of its creators.